CloudKon: a CLOUD-enabled distributed tasK executiON framework

— Task scheduling and execution over large scale, distributed systems plays an important role on achieving good performance and high system utilization. Job management systems need to support applications (e.g. Many-Task Computing – MTC, MapReduce) with a growing number of tasks with finer granularity due to the explosion of parallelism found in today's hardware which requires techniques such as over-decomposition to deliver good performance. Our goal in this work is to provide a compact, lightweight , scalable, and distributed task execution framework (CloudKon) that builds upon cloud computing building blocks (Amazon EC2, SQS, and DynamoDB). Most of Today's state-of-the-art job execution systems have predominantly Master/Slaves architectures, which have inherent limitations, such as scalability issues at extreme scales and single point of failures. On the other hand distributed job management systems are complex, and employ non-trivial load balancing algorithms to maintain good utilization. CloudKon is a distributed job management system that can support millions of tasks from multiple users delivering over 2X the performance compared to other state-of-the-art systems in terms of throughput – all with a code-base of less than 5%. Although this work was motivated by the support of MTC applications, we will outline the possible support of HPC applications as well.